Interactive Multi-Label Segmentation (web-version)

Interactive image segmentation deals with partitioning an image into multiple pairwisedisjoint regions based on input provided by a human operator. Being interactive means, that an algorithm has to quickly react on user input, which limits the computational complexity of the employed algorithms drastically. Therefore, many interactive segmentation methods represent these regions with simple models based on low-dimensional feature spaces, which in turn introduces a limitation in terms of expressibility of these models and thus segmentation quality. Furthermore, most methods can only handle the two-label case, i.e. the segmentation of an image into foreground and background. In this work, we investigate the incorporation of arbitrary high-dimensional features in an interactive multi-label segmentation framework. With such high-dimensional features, not only color and grayvalue information, but also complex textural properties of a region can be modeled. In order to not violate the runtime constraints, we carefully select the building blocks of our framework according to their ability of being implemented on parallel architectures: We employ Haralick texture features and Local Binary Patterns to represent local image structure, as well as Random Forests as learning algorithm. Finally, we employ a multi-label Potts regularizer in order to obtain spatially compact image segments. All these parts are implemented on the GPU or multi-core CPUs in order to achieve runtimes that allow for convenient user interaction. We furthermore address the problem of comparatively evaluating interactive multilabel segmentation algorithms and introduce a large novel benchmark dataset. With this dataset, we perform detailed experiments in order to evaluate the performance and runtime of the building blocks of our framework. We show the benefit of incorporating texture and color features over employing color features alone. Finally, we compare our framework to the state-of-the-art Power Watersheds method and highlight advantages and drawbacks of both approaches.